FIELD OF THE INVENTION 

This invention relates generally to teleoperator 
method and apparatus, and particularly to those which 
include means for providing the operator of remote 
apparatus with the same sense as working directly with 
his hands at the worksite. 

BACKGROUND OF THE IN VENTION 

Teleoperating, which is well known, includes the 

„ c ^ , ^ = -onnte location using 
human perrormance of ,_as^s a^. a --mooe _vj^ & 

manipulators. Telepresense includes providing the 

teleoperator with the sane feedback and control that he 

would" have were he actually at the worksite carrying out 

the operation with his own hands. Telepresence operation 

generally includes use of a stationary visual display, 

particularly a stereographic visual display of the remote 

workspace. Stereoscopic television systems are well 

known" as shown, for example, in U.S. Patent Numbers 

4,562,463 and 4,553,117 and in U.K. Patent Application 

G3 2 , 040 , 134 . 

Remote manipulators employing stereoscopic TV 
viewing together with force feedback also are well known 
as shown, for example, in an article entitled, 
"Controlling Remote Manipulators Through Kinesthetic 
Coupling," Bejczy et ai, Computers in Mechanical 
Engineering, July 19S3, ??s . 43-60, and in an article 
entitled, "Stereo Advantage fcr a ?eg-In-Hcle Task Using 
a Force-Feedback Manipulator" by E.H. Spain, SPIE Vol. 
1256 Stereoscopic Displays and Applications, 1990, ??s. 
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244-254. in the Bejczy et al article, force-torque 
feedback is disclosed. Also, in U.S. Patent Number 
2,921,445, a manipulator which includes force, torque and 
sli? sensors of a type which may be employed with the 
present invention is shown. 

Even though the operator of prior art manipulators 
is provided with a stationary three-dimensional image of 
the workspace, and manual controllers for control of the 
manipulators are provided with feedback, the operator is 
not provided with a sense of actually being present at 
the "worksite. The present invention is directed to a 
viewing arrangement for use in a remote manipulation 
system which substantially adds to the operator's sense 
of presence at the remote manipulator site. 
SUMMARY OBJECTS OF TH ~ TNVENTION 

An object of this invention is the provision of an 
improved telecperator system and method which include an 
improved viewing system to enhance the operator's sense 
of" presence at remote manipulators controlled by the 
operator from a remote location. 

An object of this invention is the provision of 
an improved telecperator system and method of the above- 
mentioned type wherein an image of manipulator end 
effectors for viewing by the operator are sensed by the 
operator as comprising an integral part of hand- 
controllers used by the operator to control the end 
effectors, thereby giving the operator a strong sense of 
oresence at the worksite. 

An object of this invention is the provision of an 
improved telecperator system and method of the above- 
mentioned type which is well adapted for use in a wide 
variety of applications including military, industrial, 

biomedical, and the like. 

The present invention includes manipulators located 
worksite and which are controlled by hand-operated 
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means at a renote operator control station. . Enc 
effectors at the manipulators are used for manipulating 
objects located in a workspace at the worksite, and 
force-torque feedback is employed for transmitting back 
to the operator mechanical resistance encountered by the 
end effectors. Stereographic visual display means 
provide the operator with an image of the workspace. In 
accordance with the present invention, the image is 
located adjacent the hand-operated means so that the 
operator locks in the direction of the hand-operated 
means for viewing the image adjacent the hand-operated 
means. Either a real or virtual image of the workspace 
may be provided adjacent the hand-operated means. 
Display means for display of a real image may be located 
adjacent the hand-operated means for direct viewing of 
the real image by the operator. For display of a virtual 
image of the workspace, a mirror is located between the 
operator's eyes and the hand-operated means. In this 
case, display means provide -a real image which is 
inverted from top to bottom, which inverted image is 
viewed via the mirror, which mirror inverts the image and 
provides the operator with a virtual image of the 
workspace, which appears to be located adjacent the hand- 
operated means. By locating the image of the workspace 
adjacent the hand-operated means the operator is provided 
with a sense that the end effectors and hand-operated 
means are substantially integral despite the fact the end 
effectors . are located at the worksite and the hand- 
operated means are located at the remote operator's 
station.- A stereophonic sound system may be included to 
Drovide the operator with stereophonic sound from the 
worksite. Video camera means are provided for viewing 
the workspace from which an image of the workspace is 
obtained. Various other sensors and associated 

resoonders may be located at the worksite and operator.' s 
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station, resoectively, for transmission of pressure, 
tactile, heat, vibration and similar information for 
enhanced telepresence operation. 

Deoending upon the application, different scaling 
nay be provided in the transmission of inf ornaLicn 
be'tween the operator's station and worksite. Tor 
example, for microassembly , microsurgery and like 
ooerations involving small part manipulation, optical 
and/or video magnification may be employed co provide an 
enlarged 3-dimensional image for viewing by the operator. 
With similar scaling between the hand operated means and 
manioulators, the perception of the operator is 
substantially that which a miniature operator would nave 
were he at the worksite. 
15 ^T?F DESCRIPTION O" THE DRAWINGS 

The invention, together with other objects and 
advantages thereof, will be better understood from the 
following description considered with the accompanying 
drawings. It will be understood that the drawings are 
20 for purposes of illustration and example only, and that 
the invention is not limited thereto. In the drawings, 
wherein like reference characters refer to the same parts 
in the several views, 

Fig. 1 is a diagrammatic shewing of a telecperator 
25 svste- embodying the present invention including side 
elevational views of a worksite and remote control 

operator's station; 

rig. 2 is an enlarged rear elevational view of the 
operator's station taken substantially along line 2-2 of 

3 0 Figure 1 ; 

Fig. 3 is an enlarged rear e ievat ior.a i view of tine 
worksite taken substantially along line 3-3 of Figure l; 

Fig- 4 is a simplified side elevational view which 
is similar to Figure 1 and showing dimensional 
35 relationships between elements at the worksite and 
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elements at the operator's station; 

Fic. 5 is a diagrammatic view to illustrate visual 
oerceotion by a miniature virtual eye, and rig. 6 is a 
diagrammatic view to illustrate visual perception by tne 
ooe-ator when image magnification is employed; 

Fig. 7 is a diagrammatic view. which is similar to 
that of Fig. 1 but showing the teleoperator system used 
for telepresence surgery; 

Fig" 3 is a rear elevational view of the operator's 

station shown in Fig. 7; 

Fig- 9 is a rear elevational view of the worksite 

shown in Fig- 7; 

Fics. 10 and 11 are fragmentary side elevational 
views of modified forms of operator's station and 
manipulator, respectively, having increased degrees of 

f reedoin ; 

Fig. I2*is a sice elevational view of a modified 
form of operator's station wherein display means are 
positioned for direct viewing by the operator; 

Fig. 13 is a rear elevational view of the modified 
form of operator's station shown in Fig. 12; and 

Fic. 14 shows a fragmentary portion of the insertion 
portion' of an endoscope for use with the present 

invention. rxPT^o /0 o r TH£ PfiE F £f><> e> £/*BA>lM£/JTS 

Refere?ce S nS ^Is made to Figs. 1-3 wherein the 
teleooerator system is shown to include an operator's 
station 20 (Fics. 1 and 2) and worksite 22 (Figs. 1 and 
3) An operator IS at the operator's station controls 
manipulator means 2 4 at the remote worksite. Manipulator 
- means 24, comprising right and left manipulators 24R and 
24L, respectively, are used for manipulating objects, 
such, as object 2 6 which is shown located on a platform, 
or base, 23 within a workspace 30 shown in broken lines. 
r c - purposes of illustration only, and not by way of 
notation, the right manipulator 2 4R is shown to 



comprise a housing 32R affixed to base 23 and fr.oir. which 
housing a telescopic am 34R extends. The inner end 34R1 
of arn 34R is mounted for pivotal movement in any pivotal 
direction using conventional mounting means. Tor 
example, the inner end of arm 34R may be mounted for 
pivotal movement about a horizontal pivot axis 36 which 
pivot axis, in turn, is adapted for pivotal movement 
about vertical axis 38. 

Arm 34R includes telescopic inner section 34R1 and 
outer section 34R2, which outer section is adapted both 
for axial movement into and out of inner section 34R1 and 
for rotation about its longitudinal axis. An end 
effector 4CR is carried at the cuter end of the arm 
which, for purposes of illustration, is shown to comprise 
a gripper. Motor means, not shown, control pivotal 
movement of arm 34R about pivot axes 3 6 and 33, axial ana 
rotary movement of outer arm section 34R2 along and about 
the longitudinal axis of the arm, and opening and closing 
of gripper 40R. The motor means, together with motor 
control circuits for control of the motors, may be 
included in housing 3 2R. The motors are under control of 
a computer 42 connected thereto through right manipulator 
interface 44R and the above-mentioned motor control 
circuits . 

The left manipulator 24L is of substantially the 
same design as the right manipulator 24R and the same 
reference numerals, but with the suffix L instead of R, 
are used to identify similar parts. For purposes of 
illustration, the left end effector 40L, shown in Fig. 3, 
is seen to comprise cutting blades which operate to cut 
in the manner of a pair of scissor blades. 

The worksite is provided with a pair of video 
cameras 4 6R and 4 6L for viewing workspace 3 0 from 
different angles for production of stereoscopic signal 
cutruts therefrom at lines 43R and 4SL. The angle y 




between the optical axes of the cameras shown in Fig. 3 
is substantially equal to the operator's interccular 
viewing angle y of an image of the workspace as shown in 
Fig. 2. 

5 The video camera outputs at lines 48R and 4SL are 

supplied to an image memory 50 for momentary storage of 
video fields of right and left: images from the cameras. 
Fields of right and left images from image memory 50 are 
alternately supplied through left/right switch means 52 

10 to visual display means 54, such as a television monitor, 
for alternate display of the two images at the face 54A 
of the monitor. Timing and control means 56 provide 
timing and control signals to various elements of the 
system, including elements included in the stereographic 

15 display system, for signal timing and control of the 
system. If digital storage means 50 are employed, then 
conversion of the camera signal outputs to digital signal 
form by analog to digital converter means prior to 
storage, and conversion of the digital signal output from 

20 left/right switch means to analog signal form in 
preparation for display at monitor 54 may be employed. 

An electrooptical device 53 at the face of the 
display means 54 controls polarization of light received 
from display means 54 under control of a left/right 

25 synchronizing signal from timing and control unit 56. 
The left and right image fields are viewed by operator IS 
wearing a pair of passive polarized glasses 60 having 
right and left polarizing elements 62 and 64 polarized in 
orthogonal directions. The polarization' of light from 

30 display 54 through electrooptical device 5S is 
synchronized field by field such that the right field is 
occluded from the left eye and the left field is occluded 
from the right eye for stereographic viewing by the 
operator. Other means for stereographic viewing of left 

35 and richt image fields are well known, including, for 





example, these using active sterecgraphic glasses, which 
-ay be used in the practice of this invention to provide 
the operator with a stereoscopic view of the remote 
workspace . 

5 The vertical deflection coil connections for monitor 

54 are reversed, causing the monitor- to scan from bottom 
to to? thereby creating a top-to-bottom inverted image 
301 of workspace 30. Letters a, b, c and d are used to 
identify corresponding corners of the workspace 30 and 

10 inverted workspace image 301. The inverted workspace 
image 3 01 is viewed by the operator via a mirror 66 at 
the top of a table 63, which mirror inverts image 301 to 
return the image as viewed by the operator to an upright 
position. Looking downwardly in the direction of the 

15 mirror, the operator views a virtual image 30V of 
workspace 30. In accordance with one aspect of the 
present invention, the image viewed by the operator, 
which in the Fig. 1-3 embodiment comprises a virtual 
image, is located adjacent controller means 70 used by 

20 the operator for control of manipulator means 24 at the 
worksite . 

Controller means 70 are shewn located beneath the 
table top 63 and include right and left controllers 72R 
and 72L for control of the respective right and left 

25 manipulators 24R and 24L. The right and left controllers 
are of substantially the same design so that a 
description of one applies to both. As with the 
manipulators, the suffixes R and L are used to 
distinguish elements of the right controller from those 

30 of the left controller. For purposes of illustration, 
and not by way of limitation, the right controller 72R is 
shewn to comprise a housing 74 R affixed to the bottom of 
table top 6S and from which hand-operated means 76R in 
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:orm of a telescopic control arm, or stick, expends. 
The right and left control arms 76R and 76L are 
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provided with the same degrees of freedom- as the 
associated manipulator arr.s 34R and 34L, respectively. 
For example, the inner end of control arm 7 6R is mounted 
for pivotal movement about a horizontal pivot axis, 
5 corresponding to manipulator pivot axis 36, which axis, 
in turn, is adapted for pivotal movement abour an 
intersecting vertical axis, corresponding to manipulator 
axis 33. Control arm 76R also includes inner section 
76R1 and outer section 76R2, which outer section is 

10 adapted both for axial movement into and out of inner 
section 76R1 and for rotation about its longitudinal 
axis. It will be apparent that the control arm 76R is 
provided with the same four degrees of freedom as the 
associated manipulator arm 34R. Additionally, sensor 

15 means 7SR are located adjacent the outer end of outer arm 
section 76R2 for use in controlling gripping action of 
gripper 40R. Similar sensor means 7SL adjacent the outer 
end of control arm "7 6L are adapted for use in controlling 
operation of scissor blades 40L. 

20 Right and left controllers 72R and 72L are included 

in a servomechanisn system wherein mechanical motion of 
control arms 76R and 76L controls the position of 
manipulator arms 34R and 34L, and pressure on sensor 
means 7SR and 7SL controls opening and closing of end 

25 effectors 40R and 40L, respectively. In Fig. 1, right 
and left hand controller interfaces SOR and SOL, 
respectively, are shown for connection of the controllers 
to computer 42. Servcmechanisms for control of 

mechanical motion at a remote location are well known, 

3 0 including those which provide force and torque feedback 
from the manipulator to the hand-operated controller 
means. Any suitable prior art servcmechanism may be used 
in the practice of the present invention, with those 
incorporating force and torque feedback being 

35 particularly preferred for telepresence operation of the 
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In the illustrated system, right and left 
microphones are included at the worksite, outputs from 
which microphones are amplified by right and left 
amplifiers and supplied to right and left speakers at the 
5 operators' station for providing a stereophonic sound 
output to provide the operator with, an audio perspective 
present at the workspace. In Fig. 1, only the right 
channel of the stereophonic system is shown including 
right microphone S2R, right amplifier S6R and right 
10 speaker SSR. The left microphone and speaker are located 
directly behind the respective right microphone . and 
speaker at the worksite and operator's control station as 
viewed in Fig. 1. Obviously, earphones may be provided 
for use by the operator in place of the speakers which 
15 would help to block out external noises at the operator's 
control station. Also, in Fig. 1 a light shield 543 at 
the monitor is shown for blocking direct viewing of the 
monitor face by the operator. 

Reference now is made to Fig. 4 wherein a simplified 
20 diagrammatic view of the system illustrated in Figs. 1-3 
is shown and wherein various lengths and angular 
positions are identified by reference characters. In 
Fig. 4, the optical path length between the cameras and 
a point F at the workspace is identified by reference 
25 character L. A corresponding path length between the 
operator's eyes and point F at the virtual image of the 
workspace is identified by the distance a-b, where a is 
the distance from the eyes of the operator to mirror 66, 
and b is the distance from the mirror to point F at the 
30 virtual image. Other dimensions shown include the height 
G of the cameras above the pivot point of manipulator arm 
34R and corresponding height g of the operator's eyes 
above the pivot point of control arm 76R. With the 
control arm 76R at length d , the manipulator arm 34R 
adjusts to length D. Similarly, with the control arm 76R 
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at an angle p A with the vertical, the manipulator arm 34R 
is positioned at the same angle from vertical. The angle 
from vertical at which the cameras view the workspace and 
the eyes view the virtual image of the workspace is 
identified by a. 

Between elements of the worksite and operator 
station, the following relationships pertain: 

a + b =kL, t 1 ) 
d = kD , and ( 2 ) 
g = kG ( 3 ) 
where k is a scale factor constant. 
When k equals 1 such that a + b = L, d = D and g = G, no 
scaling of worksite dimensions is required. 

Anv scale factor may be employed, the invention not 
being limited to full-scale manipulation. For example, 
the worksite can be small, including microscopic in size, 
in which case the optical parameters, including distance 
to object, interccular distance and focal length, and 
mechanical and dimensional parameters are appropriately 
scaled . 

3v using appropriate scaling and image magnification 
and force and torque feedback, and by locating the image 
30V of the workspace 30 adjacent hand-operated control 
means 76R and 76L, the operator is provided with a strong 
sense of directly controlling the end effectors 40R and 
40L. The operator is provided with a sense that the end 
effectors 4 OR and 40L and respective control arms 76R and 
75L are substantially integral. This same sense of 
togetherness of the hand-operated control means and end 
ffectors is not provided in prior art arrangements 
wherein the image viewed by the operator is not located 
adjacent the hand -operated control means. Even where the 
oricr art includes stereoscopic viewing and force and 
tcrcue feedback, there is a feeling of disconnectedness 
cf the hand motions from the visual image object being 
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worked upon. The present invention overcomes this sense 
of disconnectedness by locating the workspace image where 
the operator's hands appear to exercise direct control 
over the end effectors. 

For small-scale manipulation, such as required for 
surgical applications, it is desired to replicate the 
visual experience that a miniature observer would have 
were he closely adjacent the actual worksite. In Fig. 5, 
the virtual eye 90 of a hypothetical miniature observer 
is shown viewing an actual workspace. Light from a 
source at a point X, Y, Z in the actual workspace 
produces a stimulus on the miniature observer's eye 90 at 
a point identified as X'/M- In Fig. 6, an eye 92 of an 
actual operator is shown viewing an enlarged image of the 
virtual workspace produced by means of a video camera 94 
used to view the actual workspace. The illustrated 
camera includes a light-receiving lens 96 and solid state 
imaging device such as ■ a charce-coupled-device (CCD) 
array 98 where the point light source at X, Y, Z is shown 
imaged at point X L , Y L , Z i - With correct scaling, a 
corresponding light source is produced at point MX^, MY L , 
MZ ; at either the real or apparent position of the face 
of the visual display which, due to stereoscopic 
operation of the system appears to the operator to 
25 originate from point MX, MY, MZ corresponding to point 
X,Y,Z at the actual workspace. At the retina of the 
actual eye 92, a stimulus is produced at point X' at 
proportionately the same position as point X ' / M at eye 90 
of the hypothetical observer. This relationship is 
ensured by selecting a correctly scaled camera distance 
and lens focal length such that the optical magnification 
m = m/m where M is the desired overall magnification 
and M„ is the video magnification. A typical video 
magnification, M v , which equals the ratio of the CCD- 
array 93 width to the display width, is about 40. 
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Reference new is made to Figs. 7 through 9 -wherein 
a modified form of this invention is shown for medical 
use. Here, right and left manipulators 100R and 100L are 
shown which are under control of right and left 
5 controllers 102R and 102L, respectively. Elements of the 
imaging system are substantially the same as those 
employed in the imaging system illustrated in Figs. 1-3 
described above except that an enlarged virtual image 
104V of actual workspace 104 is provided for viewing by 

10 the operator. Also, servomechanism elements for 

connection of the right and left controllers 102R and 
102L to the respective manipulators 100R and 100L are 
substantially the same as those described above with 
reference to Figs. 1-3. In the illustrated arrangement, 

15 the right and left manipulators are of substantially the 
same construction as are the right and left controllers, 
such that a description of one manipulator and one 
controller applies to both. Again, suffixes R and L are 
used to distinguish between right and left elements 

20 thereof. 

The manipulators include outer control sections 
100RA and 10 OLA and insertion sections 100R3 and 100L3, 
which insertion sections are adapted for insertion into 
a body cavity through cylindrical tubes, or cannulas, not 

25 shown. For purposes of illustration, the manipulators 
are shown inserted through the abdomen wall 106 of a 
subject. As is well understood, for laparoscopic 
surgical procedures, wall 106 is separated from internal 
organs by insufflation wherein a gas is introduced into 

3 0 the abdomen by any suitable means not shown. Manipulator 
motors and associated motor control circuits are 
contained in the cuter control sections 100RA and 10CLA 
of the manipulators for control of the insertion section. 
The manipulators, together with a laparoscope 103 for 

3 5 viewina organs within the cavity, are carried by a fixed 
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^- = cr^-rr-ca" 1 table uocn which the 
r£ il no forming par" a s — g_t-<2- 

subject is supported. _• 

The insertion sections a00R3 and 100L3 or .he 
,anioulators nay be of substantially the same desxgn as 
nanioulator arms 34R and 34L described above with 
Terence to the Figs. 1-3 eminent. The insertion 
"sections are of relatively small si,e for use insice the 
bodv rnsertion section 100R3 includes telescopic mner 
seci:W 112R1 and outer section 112R2, which outer 
section is adapted for both axial movement into and out 

,. n d rotation about i<-s 

of inner section 112R1 ana 

loncitudinal axis. End effectors 114R and 114L are 
ca~*ed at the outer ends of the respective right and 
^"sections 112R2 and 112L2 for manipulation of organ 
!,6. The inner section 112R1 is adapted for pivotal 
movement about intersecting perpendicular axes locatea 
substantially at point P where the insertion section 
« -ejects wall 106. Exclusive of operation o. ena 
electors 114R and 114L the manipulator ar=s each are 

. . . ..^ ^ OUT - dec-e-s of freedom, the sane as in the 
□'ovicec wiuH - our ceu^e-a w_ 

--^ effectors 114R ana 
embodiment shown m rigs i-3. r...- e-.ecw 

_ • „ occprt^nv, microsurgical 
i-4L sir.olv may comprise, esse,. t .a..y, 

tI ,-^w h 2 -/ies removed including, for 
instruments wi^h ^he^_ na.. ^ - 

-p— ac-o-s electrosurgical cutters and 
examole, re^ac>.o-s, «__v_ ., 

~, • ^ir-n'o'ceas microneedle holcers, 

25 coagula-ors, nico.o.ceps , 

d-secting scissors, blades, irrigators, and sutures. 

La-a-oscooe 10S for viewing the workspace 104 is 
sr~wn comprising an outer operating section 10SA anc 
insertion "section 10S3. The outer end section 120 of 

in « i s a v- a nv and rotatably movable 
"0 i-serticn section luod is ^ - - ^ 

wV t Mn the inner end 122 thereof, and is provided with a 
^7 i^ace transmission windows 124,124 for 

• ^ ., n -v s ^- 104. The laparoscope 

stereoscopic viewing c_ *c..-Swei~- 

• - — p^p^s not shown .cr 

a sc is crov^e- w 

35 illuminating the workspace, and with liquid inlet ana 
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outl.t -ar.s, not shown, for flow of liquid past the 
windows. Video caaera r.eans within section 10.A a.e 
„,„ ivl t = licht received through the viewing wrnoows 
r o/c'eneraticn of !eft and right electronic ir.agej , at . 
cutout lines 43R and 43^ for connection to ™«« "-^ 
50." A unified 3-dir.ensional iuage 1041 « proaucd ». 
disolav r,ear.s 54 for viewing by the 6perator wear ng 
cross-polarized glasses 60 via oirror S6 As w! h .h 
e,bcdi=ent shown in Figs. 1-3, a virtual i B a,e »4V « 
the workspace 104 is produced adjacent control arras 30R 
and 130L of controllers 102R and 102L. Control ^ 130* 
and i30L are of the sar,e type as control arr.s 76R ana 76. 
lauded in the rigs. 1-3 e=bodi,ent described above 
-hev include telescopic inner and outer seasons 132R 
and"l32R, and 132L1 and 132L2. Sensor r.eans 134R and 
134L located adjacent the outer ends of the control ar=s 
control operation of end effectors 1M. »» 11«. 
respectively, in the manner described 
reference to Figs. 1-3. It here will be notea that the 
angle fro* vertical at which the ir.age is viewed by .n. 
ooe-atcr need not equal the angle fro. vertical at wh!=h 
the object is viewed by the car.eras. In the arranged 
illustrated in Figs. 7-9, the operator is shown to vl» 

, nnf Pn ancle 9 fron vertical (rig. 7) 

»-he i_— age 104V a>- en a.^ j-<= ^ 

whereas the object 116 is shown as viewed directly 
downwardly. With no external reference, the sense o. 
vertical within a body is not particularly great, and no 
contusion is produced in the mind of the operator as a 
result of the different observer and car-era viewing 
angles relative to vertical. 

With the Figs. 7-9 enbodinent, not only is a 
nac -^ed virtual i^.age 104V of the workspace provided 
^:'v"o-r.c bv the operator, but control arms 130R and 
::; T c _- g - e ; ter ler .gth than the length of the manipulator 
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insertion sections 100R3 and 100LB are employed. 
Servcmechanism scaling of axial movement of the 
telescopic control arms is provided such that axial 
extension or retraction thereof results in a smaller 
5 extension or retraction of the telescopic insertion 
sections. Angular pivotal motion of. the control arms 
130R and 130L produces the same angular pivotal motion of 
insertion sections 100R3 and 100L.B , and rotational 
movement of the end' sections 132R2 and 132L2 of the 
10 control arms produces the same rotational motion of end 
sections 112R2 and 112L2 of the insertion sections of the 
right and left manipulators, without scaling. This 
embodiment of the invention, with its magnified image, is 
of particular use in the area of microsurgery, and 
15 especially in those cases where the surgeon cannot reach 
an area by hand because of size constraints. 

The present invention is not limited to use with 
manipulators having any particular number of degrees of 
freedom. Manipulators with different decrees of freedom 
20 which are well known in the art may be used in the 
practice of this invention. In Figs. 10 and 11, to which 
reference now is made a controller 140 and manipulator 
14 2, respectively, are shown which include a wrist joint 
to orovide the same with additional freedom of movement. 
25 The illustrated controller 140 includes a housing 144 
affixed to the bottom of table to? 63 upon which table 
mirror 66 is located. An enlarged virtual image 146V of 
actual workspace 146 is provided adjacent the operator's 
hand 14S viewable by the operator when looking downwardly 
30 onto the mirror 66 in a manner described above. 

A control arm 150L comprising inner and outer 
sections 150L1 and 150L2, respectively, is mounted within 
housing 144 fcr pivotal movement in any pivotal direction 
as indicated by intersecting double-headed arrows 152 and 
35 154. The outer sec-ion 150L2 is adapted for axial 
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movement into and out of inner section 150L1 in the 
direction of double-headed arrow 156. It also is adapted 
for rotation about its longitudinal axis in the direction 
of double-headed arrow 153. In this embodiment, the 
5 control arm includes an end section 160 pivotaily 
attached to outer section 150L2 by wrist joint 162 for 
pivotal movement in the direction of double-headed arrow 
164. End section 160 comprises axially aligned inner and 
outer sections 160A and 160B, the outer section 160B of 
10 which is rotatable about its longitudinal axis in the 
direction of double-headed arrow 166. As with the above- 
described arrangements, sensor means 163 are located 
adjacent the free end of the control arm for operation of 
an end effector 170 at manipulator 142 shown in Fig- 11 : J 
Referring to Fig- 11, end effector 170 is shown to 
comprise a pair of movable jaws attached to a wrist 17 2 
comprising axially aligned links 172A and 172B. Outer 
link 172B is rotatable about its longitudinal axis 
relative to inner link 172A by motor means, not shown, in 
20 the direction of double-headed arrow 166M in response to 
rotation of section 160B of the hand-operated control 
unit in the direction of arrow 166. Wrist link 172A is 
pivotaily attached to manipulator forearm 174 for pivotal 
movement in the direction of double-headed arrow 164M in 
25 response to pivotal movement of end section 160 of the 
hand-operated control means about pivot axis 162. 
Forearm 174 is longitudinally axially movable in the 
direction of double-headed arrow 156M in response to 
axial movement of outer section 150L2 of control arm 150L 
30 in the direction of double-headed arrow 156. It also is 
rotatable about its longitudinal axis in the direction of 
double-headed arrow 153H in response to rotation of outer 
section 150L2 of control arm 150L in the direction of 
double-headed arrow 15S. Additionally, it is pivotaily 
35 movable about point 176 in the directions of double- 
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headed arrows 152M and 154M in response to pivotal 
movement of control arm 150L in the directions of double- 
headed arrows 152 and 154, respectively. For biomedical 
use, such as remote laparoscopic surgery, pivot point 176 
is substantially located at the level of abdominal wall 
17S through which the manipulator extends. In Fig. 11, 
manipulator arm 174 is shewn extending through a cannula 
180 which penetrates the abdominal wall. 

The outer operating end of the manipulator is 
adapted for attachment to a supporting rail, not shown, 
of the surgery table upon which the subject is supported. 
It includes an end effector drive motor 132 for opening 
and closing of gripper 170. Wrist drive motor 134 
controls pivotal movement of wrist 172 in the direction 
15 of double-headed arrow 164M, and extension drive motor 
136 controls axial movement of manipulator arm 174 in the 
direction of double-headed arrow 156M. Forearm pivotal 
control motors and linkages, identified generally by 
reference numeral 133, provide for pivotal movement of 
20 arm 174 about pivot point 176 in the directions of arrows 
152M and 154M. Pivotal motion about point 176 is 
provided by simultaneous lateral movement of the outer 
operating end of the manipulator and pivotal movement of 
arm 174. Movements are coordinated such that the cenier 
25 of rotation of forearm 174 is fixed in space at point 176 
at the level of the abdominal wall. 

Controller 140 and manipulator 142 are included in 
a system such as shown in Figs. 7, 3 and 9 which includes 
a second controller and manipulator for use by the 
operator's right hand, and associated servomechanism 
means of any suitable type, net shewn, for remote control 
of the manipulators by the hand-operated controllers. 
Video camera means at the worksite, such as shown in Fig. 
9, tec-ether with display means, such as shown in Fig. 7, 
are emolcved for erevicing the operator with an image of 
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the workspace at: a location adjacent the left and- righz 
hand-operated control neans. 5y using manipulators with 
a wrist joint, an added degree of freedom is provided for 
increased maneuverability and usefulness thereof. 
However, as noted above, the present invention is not 
limited to use with manipulators with any particular 
degree of freedom. 

Reference now is made to Figs. 12 and 13 wherein a 
modified form of this invention is shown which provides 
for direct viewing , of a 3-dimensional image 2401 of a 
workspace, not shown. In Figs. 12 and 13, only the 
operator's station is shown, which includes right and 
left controllers 242R and 242L and associated right and 
left hand-operated means 244R and 244L which may be of 
the same type as controllers and control arms described 
above. The operator's station is adapted for remote 
control of manipulators which also may be of the above- 
described type. The 3-dimensional image 2401 of the 
workspace is provided by visual display means 246 in 
conjunction with electrocpt ical device 53 at the face or 
the display means and cross-polarized glasses 60 worn by 
the operator, to which display means left and right video 
fields from left and right video cameras that view the 
workspace are alternately supplied, all in the manner 
described in detail above. End effector and object 
images 243 and 250, respectively, are shown within the 
workspace image as viewed by video cameras at the 
worksite. The display means 246 is located adjacent the 
left and right hand-operated means 244R and 244L for 
direct viewing by the operator. With this arrangement, 
the end effector and object images together with the 
hand-operated means 244R and 244L are simultaneously 
viewable by the operator. Since the hand-operated means 
also are visible, the operator is provided with a visual 
sense of connection between the end effector means and 
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hand-operated means whereby they appear substantially as 

being integral. 

Reference new is made to Fig. 14 wherein the distal 
end portion, cr tip, 260 of the insertion section of an 
5 endoscope is shown which is of substantially the sane 
type as shewn in the above-mentioned publication entitled 
"Introduction to a New Project for National Research and 
Development Program (Large-Scale Project) in FY 1S91" 
which endoscope may be used in the practice of the 

10 present invention. The insertion end of the endoscope 
includes a pair of spaced viewing windows 262R and 262L 
and an illumination source 264 for viewing and 
illuminating a workspace to be observed. Light received 
at the windows is focused by objective lens means, not 

15 shown, and transmitted through fiber-optic bundles to a 
pair of cameras at the operating end of the endoscope , 
not shown. The camera outputs are converted to a 3- 
dimensional image of the workspace which image is located 
adjacent hand-operated means at the operator's station, 

20 not shewn. Right and left steerable catheters 2 6SR and 
26SL pass through accessory channels in the endoscope 
body, which catheters are adapted for extension from the 
distal end portion, as illustrated. End effectors 270R 
and 270L are provided at the ends of the catheters which 

25 may comprise conventional endoscopic instruments. Force 
sensors, not shown, else are inserted through the 
endoscope channels. Steerable catheters which include 
control wires for controlling bending of the catheters 
and operation of an end effector suitable for use with 

30 this invention are well known. Control motors for 
operation of the control wires are provided at the 
coerating end of the endoscope, which meters are included 
in a servomechanism of a type described above for 
operation cf the steerable catheters and associated end 
25 effectors from a remote operator's station. As with the 
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other enbcdimerits, the interfacing computer in the 
servo-mechanism system remaps the operator's hand motion 
into the coordinate system of the end effectors, and 
images of the end effectors are viewable adjacent the 
5 hand-operated controllers in a manner described above. 
With this embodiment, the operator has the sensation of 
reaching through the endoscope to put his hands directly 
on the end effectors for control thereof. Endoscopes of 
different types may be employed in this embodiment of the 
10 invention so long as they include one or more accessory 
channels for use in control of end effector means, and 
suitable viewing means for use in providing a visual 
display of the workspace. For example, gastric, 

colonscopic, and like type , endoscopes may be employed. 
15 The invention having been described in detail in 

accordance with requirements of the Patent Statutes, 
various other changes and modifications will suggest 
themselves to those skilled in this art. For example, as 
noted above, the invention may include the use of tactile 
20 feedback to provide the subtle sensations for palpation 
and for manipulating tissues and instruments. To provide 
this feedback, tactile sensor arrays may be included on 
the end effectors which are coupled to tactile sensor 
stimulator arrays on the hand-operated control means, 
25 which reproduce the tactile sensation on the operator's 
hands. A variety of transduction technologies for 
telecoerator tactile sensing are known including 
resistive/ conductive , semiconductor , piezoelectric 
caoacitive and photoelectric. Hand-operated control 
30 means and manipulators of different types may be employed 
us i nc a wide variety of well-known mechanisms and 

s including, for example, 
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gimbals, linkages, pulleys, cables, drive belts and 
bancs, gears, optical or electromagnetic position 
encoders, and angular and linear motors. Force feedback 
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to 



the operator requires use of body contact with hand 
operated" control neans. 3oth hand grip type hand 
controllers such as those illustrated, and control brace 
type hand controllers are well adapted for use with the 
5 present invention for force feedback to the operator. 
Control brace hand controllers include use of structures 
with positive sensors mounted on the operator at joints 
for measuring joint angles- Force feedback then can be 
applied to each joint. Similarly, light fabric gloves 
10 with variable-resistance or fiber-optic flex sensors 
mounted on the joints for measuring bending of individual 
fingers may be used. Gloves of this type also may be 
provided with force feedback to provide for telepresence 
Interaction with real objects. Regardless of the type of 
15 hand-operated control means employed, an image of the 
workspace is produced adjacent thereto to provide the 
operator with a sense that the end effector means and 
hand-operated control means are substantially integral. 
Also, as noted above, servomechanisms of many different 
20 types are well known in the robotic and teleoperator 
system arts, and the invention is not limited to any 
particular type. Those that include force and torque 
feedback to the operator are preferred to contribute to 
a telepresence sense of operation. In addition, many 
25 different means for producing a stereoscopic image of the 
workspace are known. For example, instead of using two 
cameras, a single camera may be employed together with 
switched cress-polarizing elements in the image receiving 
path. In this case, a pair of spaced stereoscopic lenses 
are used for viewing the workspace from different angles 
and providing first and second images thereof to the 
camera. In the ?ic. S arrangement, wherein a laparoscope 
is shown, ether types of endoscopes may be used for 
viewing the workspace. As noted above, the invention is 
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limited to any particular application or use. In the 



biomedical field, uses include, for example, open 
surgery, including surgery from a remote location, 
microsurgery, and minimum invasive surgery such as 
laparoscopic and endoscopic surgery. Laboratory use 
including microscopic manipulation also is contemplated. 
Industrial use of the invention include, for example, 
hazardous materials handling, remote operations, 
microassembly, and the like. Military and undersea use 
of the teleoperator system of this system are apparent. 
It is intended that the above and other such changes and 
modifications shall fall within the spirit and scope of 
the invention defined in the appended claims. 



